Distributed and tiered architecture for content search and content monitoring

ABSTRACT

An efficient large scale search system for video and multi-media content using a distributed database and search, and tiered search servers is described. Selected content is stored at the distributed local database and tier1 search server(s). Content matching frequent queries, and frequent unidentified queries are cached at various levels in the search system. Content is classified using feature descriptors and geographical aspects, at feature level and in time segments. Queries not identified at clients and tier1 search server(s) are queried against tier2 or lower search server(s). Search servers use classification and geographical partitioning to reduce search cost. Methods for content tracking and local content searching are executed on clients. The client performs local search, monitoring and/or tracking of the query content with the reference content and local search with a database of reference fingerprints. This shifts the content search workload from central servers to the distributed monitoring clients.

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/393,971 entitled “Distributed and TieredArchitecture for Content Search and Content Monitoring” filed on Oct.18, 2010 which is hereby incorporated by reference in its entirety.

CROSS REFERENCE TO RELATED APPLICATIONS

U.S. patent application Ser. No. 12/141,163 filed on Jun. 18, 2008entitled “Methods and Apparatus for Providing a Scalable Identificationof Digital Video Sequences”, U.S. patent application Ser. No. 12/141,337filed on Jun. 18, 2008 entitled “Method and Apparatus forMulti-dimensional Content Search and Video Identification”, U.S. patentapplication Ser. No. 12/772,566 filed on May 3, 2010 entitled “MediaFingerprinting and Identification System”, U.S. patent application Ser.No. 12/788,796 filed on May 27, 2010 entitled “Multi-Media ContentIdentification Using Multi-Level Content Signature Correlation and FastSimilarity Search”, and U.S. patent application Ser. No. 13/102,479filed on May 6, 2011 entitled “Scalable, Adaptable, and ManageableSystem for Multimedia Identification” have the same assignee as thepresent application, are related applications and are herebyincorporated by reference in their entirety.

FIELD OF THE INVENTION

The present invention generally relates to information retrieval indistributed multimedia data base systems.

BACKGROUND OF THE INVENTION

Media applications which include video and audio database management,database browsing and identification are undergoing explosive growth andare expected to continue to grow. To address this growth, there is aneed for a comprehensive solution related to the problem of creating amultimedia sequence database and identifying, within such a database, aparticular multimedia sequence or sequences that match the content thatis played out with or without media content distortions. Applicationsfor such a comprehensive solution include video database mining,copyright content detection for video hosting web-sites, contextualadvertising placement, and broadcast monitoring of video programming andadvertisements.

There is a need for scalable search systems for large scale multi-mediacontent identification and monitoring. Television channel organizations,advertising agencies, and other commercial and personal interests desirea system and method for monitoring and searching of broadcast televisionshows, films, and commercials and online broadcast of various cable andinternet networks programming. Another application is monitoring andgathering statistics for large audiences viewing of various media onTVs, computers, and portable devices. Other popular applications of acontent search and monitoring system are related to improving userexperiences, enabling social communication, and various forms ofadvertising. To provide such applications, content to be identified ormonitored is compared to content stored in one or more large databasesof videos and media content. This represents a massive database searchand correlation problem. To be of value the search systems shouldsupport real time database searching, monitoring, and updating. Thesophistication, flexibility, and performance that are desired exceed thecapabilities of current generations of software based solutions, in manycases, by an order of magnitude.

SUMMARY OF THE INVENTION

In one or more of its several aspects, the present invention recognizesand addresses problems such as those described above. To such ends, anembodiment of the invention addresses a method for content monitoring.Fingerprints are generated in a user device for user watched contentthat includes popular and real time live content. The user watchedcontent is searched for in a reference database. A menu of option isgenerated based on a reference database content that matched with theuser watched content. The menu of options is displayed on the userdevice to solicit user selection of one displayed option.

Another embodiment of the invention addresses a method of performingsearch and video tracking. Content of a query generated on a user deviceis classified to generate a classified query. The classified query iscompared with classified reference contents stored in a local cacheddatabase on the user device. A menu of option is generated based on areference database content that matched from a search with the query.The menu of options is displayed on the user device to solicit userselection of one displayed option.

Another embodiment of the invention addresses a method for fast updatingof a search database. Signatures of a real time database update arestored in sequential order as received in a buffer on a user device. Thesignatures are sent from the user device to a remote database withoutlocks. A remote database is updated with the signatures of the real timeupdate.

Another embodiment of the invention addresses a method for fast updatingof a search database. Two duplicate databases are created using aninitial set of reference signatures. A first database of the twoduplicate databases is searched as the active database. A seconddatabase of the two duplicate databases as the standby database isupdated with new signatures for new content to be added to the searchdatabase. The standby database is switched with the active database tocreate a new standby database. The new standby database is updated withthe new signatures for the new content.

Another embodiment of the invention addresses a method of publishingcontent for viewing on remote client devices. Metadata that includesmenu options with links to control viewing of content on a remote clientdevice is created on a server. The metadata is distributed to the remoteclient device which overlays the menu options on a viewing screen of theremote client device in response to selected metadata. The content ispublished in response to a selection of an option from the menu options.

These and other features, aspects, techniques and advantages of thepresent invention will be apparent to those skilled in the art from thefollowing detailed description, taken together with the accompanyingdrawings and claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a fingerprinting and search system for media contentmonitoring and searching in accordance with the present invention;

FIG. 1B illustrates a tiered system with multiple servers at each tierof the search system server(s) in accordance with the present invention;

FIG. 2A illustrates a distributed client tiered system that makes use ofmultiple methods for selecting content for Tier1 Search Servers forvideo search in accordance with the present invention;

FIG. 2B illustrates a classification process of long tail videosignatures for large scale video search in accordance with the presentinvention;

FIG. 2C illustrates a process to perform query selection andoptimization in accordance with the present invention;

FIG. 2D illustrates an exemplary signature database organized by acluster key index in accordance with the present invention;

FIG. 2E illustrates an exemplary method to update a signature databaseusing replicated databases in accordance with the present invention;

FIG. 2F illustrates an exemplary method to update a cluster indexdatabase by use of circular buffers in accordance with the presentinvention;

FIG. 3 illustrates a method of performing distributed search with remoteclient content tracking and searching using centralized tiered searchservers in accordance with the present invention;

FIG. 4A illustrates a method of performing distributed search withclient content tracking and searching using centralized tiered searchservers in accordance with the present invention;

FIG. 4B illustrates a method of performing tracked search on centralizedtiered search servers in accordance with the present invention;

FIG. 5 describes a process of performing distributed content search withdistributed databases in accordance with the present invention;

FIG. 6 shows a system for content search and monitoring with popularquery caching, including unidentified query caching in accordance withthe present invention;

FIG. 7 illustrates a method of classification of query content usingquery content based fingerprints in accordance with the presentinvention;

FIG. 8A illustrates a method of organizing a search database for ascalable content search system in accordance with the present invention;

FIG. 8B illustrates a method of creating a not-identified database for ascalable content search system in accordance with the present invention;

FIG. 8C illustrates a method of organizing a search database withclassification for a scalable content search system in accordance withthe present invention;

FIG. 8D illustrates a method of managing a not-identified database for ascalable content search system in accordance with the present invention;

FIG. 9A illustrates a method of providing a customized overlay menu to aremote user for use with a remote search and tracking system inaccordance with the present invention;

FIG. 9B illustrates a signature data structure for use in remotesynchronized applications in accordance with the present invention; and

FIG. 9C illustrates an exemplary menu data structure suitable for use ina metadata based data structure for remote synchronized applications inaccordance with the present invention.

DETAILED DESCRIPTION

The present invention will now be described more fully with reference tothe accompanying drawings, in which several embodiments of the inventionare shown. This invention may, however, be embodied in various forms andshould not be construed as being limited to the embodiments set forthherein. Rather, these embodiments are provided so that this disclosurewill be thorough and complete, and will fully convey the scope of theinvention to those skilled in the art.

It will be appreciated that the present disclosure may be embodied asmethods, systems, or computer program products. Accordingly, the presentinventive concepts disclosed herein may take the form of a hardwareembodiment, a software embodiment or an embodiment combining softwareand hardware aspects. Furthermore, the present inventive conceptsdisclosed herein may take the form of a computer program product on acomputer readable storage medium having non-transitory computer usableprogram code embodied in the medium. Any suitable computer readablemedium may be utilized including hard disks, CD-ROMs, optical storagedevices, flash memories, or magnetic storage devices.

Computer program code or software programs that are operated upon or forcarrying out operations according to the teachings of the invention maybe written in a high level programming language such as C, C++, JAVA®,Smalltalk, JavaScript®, Visual Basic®, TSQL, Python, Ruby, Perl, use of.NET™ Framework, Visual Studio® or in various other programminglanguages. Software programs may also be written directly in a nativeassembler language for a target processor. A native assembler programuses instruction mnemonic representations of machine level binaryinstructions. Program code or computer readable medium as used hereinrefers to code whose format is understandable by a processor. Softwareembodiments of the disclosure do not depend upon their implementationwith a particular programming language.

The methods described in connection with the embodiments disclosedherein may be embodied directly in hardware, in a software moduleexecuted by a processor, or in a combination of the two. A softwaremodule may reside as non-transitory signals in RAM memory, flash memory,ROM memory, EPROM memory, EEPROM memory, registers, hard disk, aremovable disk, a CD-ROM, or any other form of storage medium known inthe art. A computer-readable storage medium may be coupled to theprocessor through local connections such that the processor can readinformation from, and write information to, the storage medium orthrough network connections such that the processor can downloadinformation from or upload information to the storage medium. In thealternative, the storage medium may be integral to the processor.

Systems and methods are described that are highly scalable to very largemultimedia databases. A reference multimedia database can be modified byadding a unit of multimedia content or removing an existing unit ofmultimedia content while it is being used for multimedia identification.A unit of multimedia content may be a frame or a sequence of frames of avideo, an audio clip, other multimedia formatted data, such as framesfrom a movie. A unit of multimedia content may also be a television showwithout advertisements, an advertisement, a song, or similar unit ofcommunication. The search system can be tuned to the desired speed ofmultimedia matching by centralized and distributed systems, byreplication of individual search machines or search machine clusters, byuse of a hierarchical tier of search machines and reference databases,by partitioning of reference databases, by multimedia query caching, bylocal client search methods, by client tracking, or by combinations ofthe previously mentioned arrangements. As an example, the search systemcan be implemented in a centralized client server model, or as adistributed system, or by a combination of such approaches. Also, adistributed search system may be operable on a variety of distributednetworks, such as a peer to peer (P2P) system. In addition, searchfunctions or a complete search operation may be operable at the client.

The following nomenclature is used in describing the present invention.For example, multimedia content represents any video, audio oraudio-visual content. Multimedia content may also represent a series ofphotographs or pictures, a series of audio files, or other associateddata, such as 3D video content or 4D content in which sensory feedbackssuch as touch feedback sensations are presented simultaneously withvisual and audio content.

The terms signature and fingerprint both denote the same structure of asequence of bits and may be used interchangeably. A fingerprint isgenerated to represent a unit of multimedia content using afingerprinting method that operates on the unit of multimedia content. Adescriptor denotes a bit structure that is an arrays of bits or an arraynumbers or a single number, and the descriptor is, in effect a digitaldescription of a feature or property of audio or video content at aparticular time. A signature, also known as a fingerprint, is generatedfrom a descriptor. A cluster key is a type of hash key. A cluster indexis a data structure that holds the signatures that have the same clusterkey. A multimedia signature index is a data structure that is used tohold signatures associated with a unit of multimedia content.

A number of exemplary goals of a multimedia identification systeminclude an ability to handle large capacity multimedia databases andhigh density media files. The multimedia identification system is toprovide high performance and respond with accurate media identificationwhen queried. Also, the overall design should be scalable to efficientlyhandle increasing capacity of the multimedia databases and an arbitrarylength of a query sequence.

A number of embodiments the present invention include a system for largescale video identification and monitoring. The system consists offingerprint clients and/or a tiered search system and a database ofsignatures representing the media content or object content. During aquery with content from user viewed video, a series of steps ofclassification, database searching, and correlation are performed toidentify the matching content at the central search servers or withinthe cached database.

For content identification and classification, content features need tobe extracted. For video content, image and video features can bedetected using various approaches such as key point detection across aset of filter scales, or using segmentation and contours to identify anobject. Alternatively a combination of algorithms may be used includingmotion segmentation and the above methods to provide highly accuratefeature and object detection. Signatures or video activityclassification can be derived from detected motion between frames of avideo sequence. Features of interest include color descriptors, texturedescriptors, image classifiers, activity descriptors, face detectors anddescriptors. Audio descriptors can include onset location, peaks orstrongest coefficients of log-frequency cepstral coefficients (LFCC) ormel-frequency cepstral coefficients (MFCC) and the strongest frequencybands, and changes in audio envelope. Other content features that can bedetected are audio events, audio phonemes, text in images using opticalcharacter recognition (OCR), or speech to text conversion.

The application of classifiers to the search system can improve searchperformance and capacity by orders of magnitude. The classificationdescriptors can be used to reduce the search space of each signature. Afew methods on classifying content and using classification to separatedatabases are described and the result in significant cost benefits invarious use cases of where millions of clients are supported.

Another method to optimize the search system is to use distributedsearch at the remote clients. A selected subsets of reference databasesin the distributed search system can be stored locally at each clientdevice and updated at planned intervals or, for example, in response toquerying a central search system. After a query content is identified,further tracking and new content identification can be continued on theremote client thereby avoiding queries to the central search system.With this system it is also possible to query locally first and send aquery to central server only if no content is found on local server.

Another method to optimize the search system is to organize the searchin multiple tiers with the most frequent or likely content placed in thefirst search tier. If there is no match found, then search can beperformed in the next search tiers that store less popular content. As afurther improvement, classification can be used to reduce the searchcost in the tier 2 or higher search levels, or at any search levelincluded the distributed search system on clients. Tiered search is usedto reduce search cost. If a first search reaches a small database with ahigh likelihood of finding a match, a search into the next tier possiblyhaving 100's of servers is avoided where the probability of finding amatch is only incrementally better. The tiered structure alsoadvantageously addresses update and maintenance requirements. Real timeupdates are typically confined to early tiers and high performanceservers. In most configurations described herein, the real time updatesare performed on servers with smaller databases allowing updates andmaintenance to be performed efficiently and with fewer memory managementrestrictions.

Another method addresses caching of popular and may include caching ofexpected content. The method also supports caching of popular unknowncontent to reduce search cost. Content that has been searched and notfound at any tier of a search system may be considered as unknowncontent and marked as unidentified and cached in an unidentified querycache. The unknown or unidentified content may be identified at a latertime by searching the unidentified query cache with new sources ofpopular content or even by human observation.

To provide for such needs, FIG. 1A illustrates a fingerprinting andsearch system 100 for both media fingerprinting and identification inaccordance with an embodiment of the present invention. Thefingerprinting and search system 100 includes user sites 102 and 103, aserver 106, a video database 108, a remote user device 114 with awireless connection to the server 106 and for example to a videofingerprinting and video identification process 112 operated, forexample, by user site 102. The remote user device 114 is representativea plurality of remote user devices which may operate as described inaccordance with embodiments of the present invention. A network 104,such as the Internet, a wireless network, or a private network, connectssites 102 and 103 and server 106. Each of the user sites, 102 and 103,remote user device 114, and server 106 may include a processor complexhaving one or more processors, having internal program storage and localuser controls such as a monitor, a keyboard, a mouse, a printer, and mayinclude other input or output devices, such as an external file storagedevice and communication interfaces.

The user site 102 may comprise, for example, a personal computer, alaptop computer, a tablet computer, or the like equipped with programsand interfaces to support data input and output and video fingerprintingand search monitoring that may be implemented both automatically andmanually. The user site 102, for example, may store programs, such asthe video fingerprinting and search process 112 which is animplementation of a content based video identification process of thepresent invention. The user site 102 may also have access to suchprograms through electronic media, such as may be downloaded over theInternet from an external server, accessed through a universal serialbus (USB) port from flash memory, accessed from disk media of varioustypes, or the like. The fingerprinting and search system 100 may alsosuitably include more servers and user sites than shown in FIG. 1A.Also, multiple user sites each operating an instantiated copy or versionof the video fingerprinting and search process 112 may be connecteddirectly to the server 106 while other user sites may be indirectlyconnected to it over the network 104.

User sites 102 and 103 and remote user device 114 may generate uservideo content which is uploaded over the Internet 104 to a server 106for storage in the video database 108. The user sites 102 and 103 andremote user device 114, for example, may also operate a videofingerprinting and video identification process 112 to generatefingerprints and search for video content in the video database 108. Thevideo fingerprinting and video identification process 112 in FIG. 1A isscalable and utilizes highly accurate video fingerprinting andidentification technology as described in more detail below. The process112 is operable to check unknown video content against a database ofpreviously fingerprinted video content, which is considered an accurateor “golden” database. The video fingerprinting and video identificationprocess 112 is different in a number of aspects from commonly deployedprocesses. For example, the process 112 extracts features from the videoitself rather than modifying the video. The video fingerprinting andvideo identification process 112 allows the server 106 to configure a“golden” database specific to its business requirements. For example,general multimedia content may be filtered according to a set ofguidelines for acceptable multimedia content that may be stored on thebusiness system. The user site 102, that is configured to connect withthe network 104, uses the video fingerprinting and search process 112 tocompare local video streams against a previously generated database ofsignatures in the video database 108.

The video database 108 may store video archives, as well as data relatedto video content stored in the video database 108. The video database108 also may store a plurality of video fingerprints that have beenadapted for use as described herein and in accordance with the presentinvention. It is noted that depending on the size of an installation,the functions of the video fingerprinting and search process 112 and themanagement of the video database 108 may be combined in a singleprocessor system, such as user site 102 or server 106, and may operateas directed by separate program threads for each function.

The fingerprinting and search system 100 for both media fingerprintingand identification is readily scalable to very large multimediadatabases, has high accuracy in finding a correct clip, has a lowprobability of misidentifying a wrong clip, and is robust to many typesof distortion. The fingerprinting and search system 100 uses one or morefingerprints for a unit of multimedia content that are composed of anumber of compact signatures, including cluster keys and associatedmetadata. The compact signatures and cluster keys are constructed to beeasily searchable when scaling to a large database of multimediafingerprints. The multimedia content is also represented by manysignatures that relate to various aspects of the multimedia content thatare relatively independent from each other. Such an approach allows thesystem to be robust to distortion of the multimedia content even whenonly small portions of the multimedia content are available.

Multimedia, specifically audio and video content, may undergo severaldifferent types of distortions. For instance, audio distortions mayinclude re-encoding to different sample rates, rerecording to adifferent audio quality, introduction of noise and filtering of specificaudio frequencies or the like. Sensing audio from the ambientenvironment allows interference from other sources such as people'svoices, playback devices, and ambient noise and sources to be received.Video distortions may include cropping, stretching, re-encoding to alower quality, using image overlays, or the like. While thesedistortions change the digital representation, the multimedia isperceptually similar to undistorted content to a human listener orviewer. Robustness to these distortions refers to a property thatcontent that is perceptually similar will generate fingerprints thathave a small distance according to some distance metric, such as Hammingdistance for bit based signatures. Also, content that is perceptuallydistinct from other content will generate fingerprints that have a largedistance, according to the same distance metric. A search forperceptually similar content, hence, is transformed to a problem ofsearching for fingerprints that are a small distance away from thedesired fingerprints. Embodiments of this invention address accurateclassification of queries. By accurately classifying query content, aclassified query can be correctly directed to relevant search serversand avoid a large search operation that generally would involve amajority of database servers.

Further embodiments of this invention address systems and methods foraccurate content identification. As addressed in more detail below,searching, content monitoring, and content tracking applications may bedistributed to literally million of remote devices, such as tablets,laptops, smart phones, and the like. Content monitoring comprisescontinuous identification of content on one or more channels or sources.Content tracking comprises continued identification of alreadyidentified content without performing search on the entire database. Forexample, a television program may be identified by comparing a queriedcontent with content already identified, such as television programs andprimarily with the anticipated time location of the program. Also, whena query is sent to a search server, prior matching information that isknown and associated with the query is also sent along with the query.Thus, the search server knows previous match time and can extrapolate topredict the likely reference time. If the server has additionalinformation to modify the predicted time, for example to subtract timefor an advertisement insertion, this is performed at the trackingfunction. A time line indicates predicted points in time. For example,if a match is made with a reference time 0:01:00 in a video clip, forexample, to a remotely viewed content at time 10:00:00, then at time10:02:00 in the remotely viewed content a match would be expected at0:03:00 in the reference content in normal circumstances. In case anadvertisement is to be inserted in the remotely viewed content, threepossible approaches may be used. In a first approach, advertisements aredetected and analyzed for duration in order to avoid searching for thesame advertisements. In a second approach, by knowing the exact time ofa shift in content to an advertisement, a query is performed at thepredicted time and a different advertisement clip may be inserted in theremotely viewed content with adjustments made if there are differencesin duration. In a third approach, by knowing the range of expectedreference time to be within the 0:01:00 to 0:03:00, a search operationon the reference can be limited to the time range 0:01:00 to 0:03:00.This is in contrast to a number of current approaches that involve alarge number of database servers for such applications.

FIG. 1B illustrates a tiered system 120 with multiple servers at eachtier of the search system server(s) in accordance with the presentinvention. The tiered system 120 is used to reduce search cost for alarge scale content monitoring and identification system. The tieredsystem 120 includes a content analysis function 121 which generates acontent query 122 that is entered in the tiered system 120 having tier 1search servers 123 and tier 2 search servers 128. Each tier 1 and tier 2search server comprises a tier 1 database (DB) 124 and a tier 2 DB 129,respectively. The tiered system 120 further includes a query resolutionunit 126 and a presentation unit 131 for presentation of final results.The most likely content is placed in the tier 1 DB 124 with fewerservers and a smaller database. Likely content is identified bystatistical methods or as identified by an operator. The likely contentincludes popular live broadcast programs and popular content such asrecent TV shows, movies, and the like. Thus, most of the queries areidentified by this small set of tier 1 search servers 123, avoidingsearch for the most likely content in the much larger tier 2 searchservers 128 and DB 129 and thus reducing search cost. If the content isnot found in the initial tier 1 search, the query is distributed orselectively sent to the tier 2 search servers 128. Additional tiers canbe used and each tier can have a plurality of servers, and the number ofservers is based on the size of database and search performancerequired. The tiered search system 120 uses various selection methods toselect content for tier1. A content query, 122 is first applied to tier1search server 123 which performs search on its content database 124. Theresult of the query in the tier 1 DB 124 is presented on path 125 to thequery resolution unit 126. The query resolution unit 126 which oftenresides in a cluster node that aggregates results from a set of searchservers, generates a final search result for presentation on thepresentation unit 131. If no match was found in the query search in thetier 1 DB 124, the query resolution unit 126 applies the query, on path127 to the next search tier, tier 2 search server 128. The results oftier2 search are sent over path 130 to the query resolution unit 126,and final results are presented on the presentation unit 131.

FIG. 2A illustrates a distributed client tiered system 200 that makesuse of multiple methods for selecting content for Tier1 Search Serversfor video search in accordance with the present invention. Contentanalysis including fingerprinting is performed in client device 201. Thedistributed client tiered system 200 provides remote clients the abilityto identify content and receive applications that are delivered bycontent owners related to the identified content. Remote clientapplication can be embedded in client devices, such as home televisions,set top boxes, buddy boxes, smartphones, tablets, laptops, and otherportable and desktop computer systems. The capacity of the searchdatabase associated with remote client application located on each ofthe client devices may vary as permitted by its compute and memoryresources. Thus, the remote client application and the distributedclient tiered system 200 is adaptable to the resources available on theplurality of remote devices, such as client device 201. The system 200is designed to allow various distributions of the search computations atthe client devices and at the centralized servers, which may use amultiple tiered system as described with FIG. 1B. The U.S. patentapplication Ser. No. 12/141,337 filed on Jun. 18, 2008 entitled “Methodand Apparatus for Multi-dimensional Content Search and VideoIdentification” discusses some methods of fingerprinting that may beused on the plurality of client devices, such as client device 201.

In another embodiment, multiple selection methods are used to selectcontent for tier1 search server(s), and for the next tier of searchserver(s). The content selection methods include selections made ontemporal periods, such as a day of the week and or time of day, forexample, geographical location, such as state, city, or street, forexample, and also may be based on a calculated value of the content,such as, most popular televised content during Sunday afternoon.Temporal selection includes real time expected content, such as currentbroadcast content, short term popular content, such as shows broadcastwithin a day or days; and longer term popular content from movies, andpreviously recorded shows. Temporal selection may also include contentthat matches more frequently among recent queries and can also includefrequent queries that do not match content in databases. Temporallyselected content can vary as per the “interests” of the day, and in atiered system reference content signatures can be moved or copied fromlower search server(s) to the higher search server(s).

Content may also be partitioned as per the geographical distributionbased on audience data and source of content. Similarly, for every user,geographical content of interest can be identified and this data can beused to reduce search cost on a tiered system having a large number ofsearch servers especially at the lower search tiers. Content having ahigh calculated value can use less selective partitioning for queriesand hence can be placed into a tier1 database which may use nopartitioning or less selective partitioning.

The output of content analysis in the form of fingerprints is sent as acontent query to a query distribution unit 202 of the search system. Thetier 1 search servers 203, uses an expected broadcast content database204, a popular short term content database 205, and a longer termpopular content database 206. Expected content refers to therelationship of queries to content in DB. Based on statistics, 80%people watch live television, for example as selected from an electronicprogram guide (EPG) and this forms the “expected content” that is storedin database 204. In summary, the expected broadcast content database 204includes selected real time broadcast content. The “popular short termcontent” database 205, illustrates the selection of most popularprograms viewed in the recent day(s) for tier1 search. Short termcontent typically refers to content that is viral or popular within thelast 24 hours. The database 205 stores popular short term content asidentified by statistics engine in the search system. Popular mediaprograms, viral content and the like are stored in the database 205. Thelonger term popular content stored in database 206, represent contentthat is popular in the recent weeks or a longer time frame and includespopular movies. For example, the database 206 stores longer term popularcontent that is steadily watched by people such as movies or certainshows. It appreciated that there may be some overlap between thesegroupings. Above all these groupings, the query distribution unit 202exists to make it easier to decide how to optimize the search cost bydistributing the content. The query distribution unit 202 isconfigurable to separate and distribute content in the tiered system inan efficient and organized manner. If no match is found at this tier,the query is sent to the next tier, at step 202 on path 207. The maingoal tiered system 200 is to distribute content in such a way as toreduce search cost without reducing accuracy.

In another embodiment, content from tier2 can be selected to be moved totier1, and vice-versa. This decision to move content can be based onupdated measures of the popularity of the content at the given searchservers.

FIG. 2B illustrates a classification process 210 of long tail videosignatures for large scale video search in accordance with the presentinvention. One embodiment is based on the application of classifiers tothe search system to improve search performance and capacity by ordersof magnitude over a standard search system for large databases that didnot use classification. The classification descriptors can be used toreduce the search space of each signature. The search space reductioncan be applied at any of tiered search levels. In another embodiment,search space reduction is achieved by translating classificationinformation into an encoded bit form. The encoded bit representation ofthe classification information is used to speed up search operations, byutilizing these encoded bit representations as index lookup addressesfor quick access to both query and reference information, for example.These encoded bit representations are a compact form of informationuseful for a targeted search in particular databases. This method relieson further sub-dividing the database for first search step by using theclassification bits in the index.

In another embodiment, classification of queried content is used toreduce search workload and may also be used to expand informationstorage in a reduced capacity database in case of local search onclients. In this method, classification data, which includes signaturesof popular content and extracts of content, are stored at the clientnodes. When a close match is found for any content, more details of thecontent may be downloaded to the client and detailed content monitoringmay then be performed at the client. The size of a contentidentification database can easily include millions of hours of contentin a system having a plurality of servers and a composite very largedatabase to be searched, resulting in a large search cost that canincrease significantly as more content is added. It is more efficient toclassify the content of a query and deliver the query to the most likelydatabases so as to avoid searching the entire database content. FIG. 2Billustrates a process that uses tiered search and classification of longtail video signatures to reduce search costs. A long tail signature is asignature which reaches the leaf nodes of a tiered system and in a wellorganized tiered database, the long tail signatures occur with lessfrequency compared to the majority of search query signatures. Multiplemethods are used to select content for searching at tier 1 searchingstep 213 tier1 search servers 214-216. The search servers host thespecified DBs. Content analysis including fingerprinting is performed atstep 211 to generate a query which may include a class or featuredescriptors of the query and a user profile. The query class pertains toclassification description generated by remote client. For example, aclassification description may include program type such as sitcoms,sports game “baseball”, program source, for example local and nationalnews programs and the like. An advantageous use of classification is tobe able to identify with high accuracy a likely matching database, whiletargeting the query to a small section of the identified database. Theuser profile refers to typical user behavior and likely viewingstatistics. For example, a typical user profile may include a listing ofidentified popular programs watched and at what time. The creation ofthe user profile can also separate out which programs are watched oneafter the other, roughly predicting what different individuals maywatch, and at what time. The viewing statistics may include actualstatistics of viewing such as which programs watched at what times, andwhat are the programs watched one after the other. The viewingstatistics are used to described user behavior and create user profile.The step 211 can also include query analysis and query optimization asillustrated in FIG. 2C and described in further detail below.

In FIG. 2B, a query distribution step 212 sends the query to be searchedat tier 1 searching step 213 in one or more of the servers 214-216. Thetier 1 searching step 213 may be directed to access server 214'sdatabase which includes selected or expected real time broadcastcontent. The tier 1 searching step 213 may be directed to access server215's database which includes popular content. The server 216's contentdatabase includes “not found queries”, which is a destination databasefor most frequent “not found queries”. The query resolution step 217evaluates the results from tier1 servers 214-216 and then decideswhether to send a query to one or more tier2 servers. The classificationstep 218 performs a final classification of the query using attachedquery class descriptors, and the attached user profile sent by step 211.The classification descriptors indicate major classification aspectswhich may include content source, major performers, content type, speedof action, nature of audio, such as silent, speech, sound effects andthe like. The classification step 218 determines likely databases, sothat the query can be sent only to the likely search servers that holdthe relevant databases to be searched for the query. For example, theclassification step 218 uses attached descriptors of the query class andcompares them with the class descriptors of each search database todetermine a list of search databases that are close to the query class.The classification step 218 identifies the search servers that aresimilar to the query. This classification is advantageous and is used todetermine to which specific search servers the query is delivered thusimproving search performance. Query distribution step 219 receives thematching list of search databases and limits the search to one or morespecific servers. Alternately, the classification of query content andsignatures can be modified by additional cluster hash index bits andthen used in a similarity search step for each query signature. Thesearch space reduction resulting from the query distribution step 219reduces the search space for each similarity search per query. The U.S.patent application Ser. No. 12/141,337 filed on Jun. 18, 2008 entitled“Method and Apparatus for Multi-dimensional Content Search and VideoIdentification” included herein in its entirety, describes contentsearch methods using search space reduction. The tier 2 searching step220 may be directed to access server databases 221-223. The database 221stores, for example, classified videos based on baseball classdescriptors. The database 222 stores, for example, classified videosbased on classical music class descriptors. The database 223 stores, forexample, classified videos based on talk show class descriptors.

The multi-media content could be classified in many ways includingclassification based on channel logos for television content or studiosin case of movies. Other classifications can be based on learned ordesignated classes and may use various feature descriptors to representeach class. An example of a classification steps at the client isdescribed and illustrated in FIG. 7.

In another embodiment, classification can be used at every search levelincluding client device search operations and each tier's searchoperations. A method of content analysis, fingerprinting and queryingincludes the following steps:

a) analyzing content to extract content features and use the extractedfeatures to classify signatures and the content itself;

b) generating fingerprints for content; and

c) checking the fingerprint before issuing a query to the search system.If information content in the query fingerprint is low than probabilityof a match may also be low, the query check function looks foradditional signatures that meets the information content requirementwithin a predetermined time limit or send the best fingerprint samplefound. Latency and response requirements of a system may be used orestimated to specify the predetermined time limit for a query and searchresponse operation.

FIG. 2C illustrates a process 240 to perform query selection andoptimization in accordance with the present invention. The process 240checks if the information content in the query is sufficient to producean effective query. In addition, optimization could be done to the queryto remove redundant content and signatures that tend to be repeated. Thequery video's signatures are accumulated in step 241 until sufficientquery information is determined to be available to perform a searchrequest. For example, unique signatures, those signatures that areseparated from other signatures by a minimum distance, are counted. Ifthe count of the total number of unique signatures during a specifiedperiod exceeds a threshold, it is determined that sufficient queryinformation exists. At step 242, the number of query signatures orunique signatures are counted or the information content of the query ismeasured. The information content of a query may be determined by anentropy based information measurement on the query content. At step 243,a decision is made if enough query information is available bythresholding the count or the information entropy measurement, forexample. By performing this thresholding check at step 243, a very highchance of query identification is assured and this in turn reducessearch costs as well as reduces delay in identification. If a thresholdis exceeded, then a search request are initiated at step 244. If thequery information is equal to or less than the threshold then more querysignatures are collected till sufficient unique signatures areavailable.

As discussed above, a cluster key is a type of hash key. A cluster indexis a data structure that holds the signatures that have the same clusterkey. A multimedia signature index is a data structure that is used tohold signatures associated with a unit of multimedia content. Generally,every signature is associated with a cluster key, which is alsoconsidered a hash key. Using such a cluster key, a cluster index may becreated which is a hash index. The cluster index is a hash datastructure where the signatures are stored at a location according thecluster key. The signatures may be stored in a flat array, in a linkedlist, or in some other data structure.

FIG. 2D illustrates an exemplary signature database 250 organized by acluster key index in accordance with the present invention. Thesignature records for the multimedia content that are to be placed intothe signature database 250 are collected together and grouped by acluster key. At this stage of processing, the number of signatures thatbelong to particular cluster key is known so the memory space for thesignature records can be allocated and the signature records may bestored in the memory. For example, a cluster key array 252 stores oneelement for each possible cluster key. The index into the cluster keyarray 252 is the integer interpretation of the cluster key as a binarynumber. Thus, given a cluster key, direct addressing into the array 252retrieves the number of matching signatures and where correspondingsignature records are located, such as a link reference address (LRA) toa list of signature records in signature record arrays 253 or 254. InFIG. 2D, for example, cluster key 255 links to the array of signaturerecords 253, and cluster key 256 links to the array of signature records254. Each entry in cluster key array 252, such as entries 255 and 256,have an additional field included in the entry that stores the linkreference address (LRA) to a signature record array, such as signaturerecord 253 and 254. The signature records stored in the memory 253 and254 are not considered fixed and unchangeable and dynamic updates to thesignature records may added as described in more detail below.

When new video content is added to a search database, existing clusterindexes need to be updated with the new video content's signatures byplacing the new signatures at appropriate locations in signature recordarrays indicated by the cluster keys associated with these signatures.When updating video content that is currently being broadcasted, such asreal time live content, the search database needs to be updatedcontinuously. U.S. patent application Ser. No. 13/102,479 titled“Scalable, Adaptable, and Manageable System for MultimediaIdentification” filed 6 May 2011 describes these updates as real-timeupdates. In a tiered search database, the first tier, where the realtime live content would most likely reside, receives most of the queryload. Hence, the first tier generally needs to be designed to performfast updates. In the U.S. patent application Ser. No. 13/102,479,methods are described to carry out such updates using exclusive locksaround the cluster index data structures. In another embodiment, amethod is described herein that does not involve locks for such updatesof the cluster index data structures.

The methods below can be used in concert with an existing database whichmay not be modified by real time updates. For example, fast updates canbe performed on a separate database of indexes which does not disturbcontent in the existing database such as popular content but notnecessarily live.

For cluster updates in a live content database, a maximum number ofsignatures for each cluster key is calculated and based on thiscalculation, space for the signatures associated with the cluster keysis pre-allocated. This maximum size can be determined by various means.For example, by design requirements and constraints, the total number ofcontent viewing hours supported by the first tier is known. Using ameasured number of signatures per second rate, an average number ofsignatures associated with each cluster index may be calculated. Thisaverage number of signatures is then modified to take into accountstatistical variations in the number of signatures in each cluster andto take into account the size of the reference database to determine themaximum number of signatures, as used herein.

A first method of implementing database for fast updates and withoutusing locks uses two databases for fast updates. These two databases arein addition to databases that are not to be changed. FIG. 2E illustratesan exemplary method 260 to update the signature database usingreplicated databases in accordance with the present invention.Initially, before the search system starts, two databases are createdusing the initial reference signatures. Thus initially these twodatabases are duplicates of each other. A search method uses a selectedone of the two databases to search for content matching the searchqueries. This selected database is called an active database while theother database is called as a standalone database. At step 262, adetermination is made whether a new update has been received. If no newupdate is received, the process 260 returns and waits until a new updateis received. If new update has been received, the process 260 proceedsto step 264. At step 264, the standalone database is updated with thenew signatures associated with the new update. At step 266, when theupdate operation is finished, the two databases are switched, such thatthe standalone database becomes the active database and the activedatabase becomes the standalone database. At step 268, the newstandalone database is also updated with the new update and the activedatabase is used by the search algorithm for the next update received.The process 260 is termed as a ping-pong implementation.

A second method of implementing database for fast updates and withoutusing locks makes use of circular buffers. FIG. 2F illustrates anexemplary method 270 to update a cluster index database by use ofcircular buffers in accordance with the present invention. A circularbuffer, having a capacity appropriate for the maximum number ofsignatures calculated to be generated within a specific time, isallocated for each cluster key and used to store the signatures. Thecircular buffer data structure can be implemented within a section ofmemory. Specifically, for example, a section of a memory that has anidentified beginning location and an identified end location can beconsidered as a circular buffer. The buffer is filled starting from thebeginning location and is full when the end location is filled. Then forthe next entry, the previous entry at the beginning location isoverwritten because it is a circular buffer and further updatesoverwrite the next sequential entries. Thus the overwritten entries arelost. The method 270 is designed in such a way that by the time theentries are overwritten another copy of the content signatures has beencreated somewhere in the search system. For example, since the previoussignatures which may be overwritten are likely to be for a lapsed event,these previous signatures may have been placed in another search tiersince there are probably much fewer viewers watching the lapsed event.Additionally, even if the likelihood of overwriting reference content islow, the likelihood of missing a possible match due to a singleoverwritten signature is much lower. Hence a system with this fastupdate method will be able to detect a target likely reference contentwith very low probabilities of missing a possible content match.

It is noted that there a number of reasons the capacity of a circularbuffer may need to be changed. For example, as the capacity of thereference database increases, the size of the circular buffer may needto be increased as well. In one embodiment of the present invention, amonitoring approach is utilized. For each signature that is overwrittenin the circular buffer, a time difference between the time stamp of thenew signature and the signature being overwritten is calculated. If thisdifference is less than a predetermined amount, a counter associatedwith that cluster key circular buffer is incremented. If the counterexceeds a predetermined threshold, the capacity of the circular bufferis increased again by a predetermined amount. After a circular buffer'scapacity has been changed, the counter is reset. The counter may also bereset for other situations.

In FIG. 2F, at step 271, a determination is made whether a new realtime-update is received by the search system. If no real time update isreceived, the process 270 ends until a new real time update is received.If a real time update is received, the process 270 proceeds to step 273.At step 273, the signatures and associated cluster keys are read. Atstep 275, the signatures are stored into the corresponding circularbuffers associated with the cluster key. The signature is always storedafter the location where the last signature in this cluster was stored.Because the buffer is circular, if it is full, then a newly readsignature is stored at the beginning of the circular buffer. Thus thesignature that was written at the start of the buffer would be replacedby the new signature. At step 277, the process ends.

By providing an adequate storage capacity in the circular buffers, themethod 270 doesn't generally need to calculate the total number ofsignatures in the update file and check if there is enough space tostore these signatures in the signature buffer. With adequate capacityin the circular buffers, this update method 270 can be implementedwithout any read/write locks. The speed up in the updating process inthis architecture is obtained at a risk that some overwriting of contentsignatures may occur. However, such risk is minimized by havingsufficient capacity that by the time entries are overwritten, anothercopy of the content signatures is created somewhere in the searchsystem.

FIG. 3 illustrates a method 300 of performing distributed search withclient content tracking and searching and using centralized tieredsearch servers in accordance with the present invention. The distributedsearch and video tracking are performed on a client's media databaseusing correlation and local cache searches based on an index lookup. Anindex lookup may be performed using an index based on a cluster hashgenerated from content descriptor and associated data and another indexmay be used based on content id, time, and feature class. The clientdevice 301 is configurable to execute a fingerprinting and analysis step302 and content tracking and content search step 303 utilizing adatabase of content descriptors and signatures 307. Streaming video 308received at the client device 301 which is operated on by the clientfingerprinting and analysis step 302. Fingerprints that are generatedare sent as a query 305 to the video search system 310. The video searchsystem 310 includes tier1 search step 311 that searches one or more ofcontent databases 313, 314, and 315. If no result is found in tier 1search, then the query 305 is sent in step 316 to tier 2 search step312. If a matching result is found in the search system, the searchresult 317 is indicated and sent on a path 306 to the client device 301and specifically to the client tracking module 303. The referencesignatures are downloaded from the search server(s) to the remote clientvia downloading step 309.

FIG. 4A illustrates a method 400 of performing a distributed search withclient content tracking and searching using centralized tiered searchservers in accordance with the present invention. Streaming video 402 ata client device feeds into the fingerprinting step 403 of the clientdevice. The generated fingerprints 404 are stored locally on the clientdevice. Each client device may receive a different stream of video andthus may generate a different set of fingerprints that are storedlocally. The generated fingerprints are used to query the search system.For example, after a match is detected, fingerprints may be trackedlocally and a query to a centralized server may not be necessary. Atstep 405, a client device's client query module sends a query on path406 to a centralized video and content search server which is searchedat step 408. The video and content search server(s) accesses video andcontent indexed databases at step 409. If the search server detects amatch, it is returned as the match result 407 which is sent to theclient device. The match result 407 is used to initiate the trackingfunction at step 415. The search server at step 410 makes adetermination whether to download signatures from the search servers tothe client device. If there is no match, then at step 410, a query canbe sent to the next search tier, if any, and if this is the last tierreport not identified. At step 411, the matching video fingerprints aredownloaded to the client device, and the signatures are stored locallyon a client database 412. The purpose of the client database is toperform continual tracking of content at the client. Alternatively acache of downloaded reference signatures could be stored in advance onthe client device. At step 415, the tracking function continues matchingthe content playing on the client to the downloaded reference. Thetracking function utilizes knowledge of the matching points identifiedby search at the central search system, to predict and evaluate thelikely matching locations for currently played content on the remoteclient. The search sequence tracking function at step 415 accesses theindexed fingerprints via step 414. When the sequence tracking functionloses track of the match between query and reference videos the trackingfunction initiates a local find function at step 417 via a find command416. At step 417, the local find function accesses the video databasevia step 413. At step 417, the local find function reports if it finds amatch to the tracking function. This is communicated to the client querymodule, which at step 405, determines whether to continue searchinglocally or to send a new query on path 406 to the centralized video andcontent search server which is searched at step 408. The ability toperform sequence tracking on a client device can reduce the search loadon central servers significantly. This is advantageous for reducingsearch time and over all searching costs. Additionally, the latency foridentification can also be reduced. Advanced methods such as scenechange detection or reduced sampling of signatures can be used to reducethe size of fingerprints downloaded to perform tracking search. It isnoted that a scene change is an event where video content changessignificantly and can be identified by finding a local peak in resultsfrom filtering images or audio content.

While tracking the matched query at the remote client the downloadedfingerprints can be sampled to reduce the downloaded fingerprint size.An adaptive thresholding technique is used with lower threshold fortracked content based on statistically learned models. For example, astatistical model indicates that for a match with a 95% confidence ofbeing a correct match a threshold is required that is at least 20% ofthe maximal score. The U.S. patent application Ser. No. 12/141,337 filedon Jun. 18, 2008 entitled “Method and Apparatus for Multi-dimensionalContent Search and Video Identification” and incorporated by referenceherein in its entirety describes how a threshold can be derived for agiven query. For a tracking query, as described herein, a relativedifference between two thresholds, one for time t1 and another for timet1+t2, can be used to determine a new threshold for a tracking queryover time t2. For example,ThresholdTrack(t2)=Threshold(t1+t2)−Threshold(t1); whereThresholdTrack(t2) is the threshold for tracking query over time t2,Threshold(t1+t2) is the threshold for detecting a match over time t1+t2,and Threshold(t1) is the threshold for detecting a match over time t1.

FIG. 4B illustrates a method 420 of performing a tracked search oncentralized search servers in accordance with the present invention. Thequery in this case indicates whether a previous match has been detectedand this is a continued query. The query also includes a matchingreference query time and a reference content alignment.

Streaming video 422 at a client device feeds into the fingerprintingstep 423 of the client device. The generated fingerprints 424 are storedlocally on the client device. Each client device may receive a differentstream of video and thus may generate a different set of fingerprintsthat are stored locally. The generated fingerprints are used to querythe search system. At step 425, a client device's client query modulesends a query via step 426 to a centralized video content search serverwhich is searched at step 428. If the search server detects a match, itis returned at step 431 as the match result which is sent to the clientdevice. In further queries where a previous match is indicated, the step426 sends the query to the tracking function at step 430. The trackingfunction utilizes knowledge of the timing alignment identified by searchat the central search system, to predict and evaluate the likelymatching locations for currently played content on the remote client.The search sequence tracking function at step 430 accesses the indexedfingerprints in the central database at step 434. If the sequencetracking function at step 430 loses track of a match, as indicated atstep 427, between a query and reference videos, it can initiate at step435 a search on the central database at step 434. At step 431, the matchresult coming from the tracking function or the central server iscommunicated to the client.

FIG. 5 describes a process 500 of performing distributed content searchwith distributed databases in accordance with the present invention. Theprocess 500 supports an option of performing search locally on remotedevices and deciding to query a centralized search server when contentis not found locally on the remote devices based on certain conditionsbeing met. If the query that is not identified matches rules for certaincontent types, then a server search may not be initiated. For example,the query may match an advertisement classification, not found locallyon the remote device. Streaming video 502 at the client device feedsinto the fingerprinting step 503 of the client device. Each clientdevice may receive a different stream of video and thus may generate adifferent set of fingerprints that are stored locally. The generatedfingerprints 504 are stored locally on the client device and these areused to form a query. These generated fingerprints at client device canbe used to query the server or to track or search locally within theclient device. At step 508, a distributed controller controls theprocess of querying at a centralized search server or doing searchoperations locally. In one embodiment, the distributed controller firstperforms a search locally on one or more client devices, and evaluatesthe classification of the query and based on the result of theclassification decides whether to query the centralized search server.At step 507, a client query module sends the query received from thedistributed controller to the centralized search server. The clientquery module sends the query on path 531 to the centralized searchservers where the query is searched at step 520. If a match is detected,a match result 532 is returned to the distributed controller for use atstep 508. At step 535, the distributed controller then requests adownload of matching video fingerprints 533, for local caching on theclient devices. The distributed controller module converts thedownloaded fingerprints into two indexes for easy access to searchfunctions. One fingerprint index is called the cluster DB index, whichis generated using the signature descriptor and associated data locallyon the client devices or may be downloaded from the centralized serverto the client devices. The other index is called the video DB index andis generated using the video content id and reference time of the videocontent locally on the client devices or may be downloaded from thecentralized server to the client devices. The cluster DB index is usedto access cluster DB 509, and the video DB index used to access video DB510. The downloaded cluster DB index is further processed to generate aconfigurable length local cluster index. The main search functions aresequence track and local find 515, which access the indexedfingerprints. Sequence tracking continues matching of identifiedsequences, by evaluating signatures at the predicted time location ofthe reference signatures, according to a time line of the referencecontent. If tracking is lost, no more matching with expected, then alocal search operation is performed. The distributed controller controlsthe search function and the complexity of searching in selected databasesearch space through command on path 512 and receives results of searchoperations through path 513.

FIG. 6 illustrates system 600 for content search and monitoring withpopular query caching, including unidentified query caching inaccordance with the present invention. The system 600 comprises a clientdevice 601 for generating a query, a centralized server(s) 602, a datalink 603 to a video database 604 associated with the centralizedserver(s) 602. The centralized server(s) 602 include an found query (FQ)cache 606 acting as a first mini-database and an unidentified query(UIQ) cache 607 acting as a second mini-database. The FQ cache 606stores the content that matches frequent queries. The UIQ cache 607stores frequent queries that are not identified in the database. Theunidentified query content from previous searches is added to a“unidentified database” in the UIQ cache 607. Query content resides inthe UIQ cache 607 to be used by the centralized server(s) 602 in newsearch operations to identify popular “undetected content”, until suchcontent is aged out of the “unidentified database”. If queries withsimilar content are repeated then the count of repeat viewing of eachunidentified content is updated. The “unidentified query database” isthus used to generate popular “unidentified queries” which will residein the search tier 1 servers. The management of the “not found query”database includes methods to age out queries that become infrequent, andmethods to remove “not found queries” when newly added content matchesthe query in the “not found query” database. Deletion from “not foundquery” occurs when addition of new content to search database is added.

Aging out of cached signatures is performed by ordering the cachedcontent by the count of recent accesses and removing content that isaccessed less often. A count of recent matches to each cached content ismaintained and used to determine which cached content is to be removed.

The above method will reduce the query load cost on the search servers.If this method was not available, then the search cost can still besignificantly large when significant fractions of the queries are notidentified. For example, the queries (signatures) are selected forcaching so as to reduce the search load on the tier 2 and lower searchservers. If a significant portion of content that is watched on theclient device 601 is not in the searched video database 604, such as 30%of the content watched, then these 30% queries will be added to the nextsearch tier. However, if the more frequent queries that are not foundare identified and further queries are avoided into the rest of thesearch system's other tiers, the percentage of queries to the remainingand largest part of the search servers can be reduced to say 10% insteadof 30% in absence of this method. This represents a very significantgain in search performance. For example, if the subsequent search tierhas 100× larger DB than the previous tier, then 30% of queries into thistier requires 30 times more search cost than if the query could havebeen applied to the first stage. Thus the cost of searching in thesecond tier, in this example, is the dominant search cost. By reducingthe % of queries going to the next tier, in this case 30% to 10%, themethod described herein provides a major search cost reduction (3× inthis case).

A query popularity analysis process is used for queries that areidentified. The identity of matching content and its frequency ofquerying are stored, and when the minimum matching queries and the rateof querying for a specific content is greater than an establishedthreshold, the signatures for the identified content are selected to bein the database or as temporary cache of first or higher tiers of thesearch servers.

In contrast the U.S. patent application Ser. No. 13/102,479 titled“Scalable, Adaptable, and Manageable System for MultimediaIdentification” filed 6 May 2011 describes “caching” with reference toFIGS. 15A-15C at pages 53-56 using methods of creating a hash index foreach popular queries that are found.

FIG. 7 illustrates a method 700 of classification of query content usingquery content based fingerprints in accordance with the presentinvention. The following exemplary set of classification methods may beused to classify the video clips at the remote client device.Classification on a combination of detected features and descriptorssuch as color descriptor, motion activity, detected objects, includingfacial recognition, object action, background translucency,reflectivity, and color, logos in video, type of sound, such as voice,type of music, chatter, loud, soft, silence. Each video's assignedclassification parameters are compared against a set of classes, whichcan be defined at the central system or can be generated based on userprofile. Class based on user profile is one possibility. In some casesit can be useful. For example, if a user is known to watch baseballgames, and either the timing for a query coincides with baseballprogramming or some features in the query is associated with baseball,then the search system may still select baseball as one likely class forthis user since a higher weight would be attributed to baseball for thisuser, based on his user profile. If the distance measured between avideo clip's classification parameters and the values of each class'sparameters are within a specified threshold, then the video isconsidered a member of that class.

Classification for search partitioning by servers is used to reduce thesearch cost. Classification may be used to direct the video query tospecific second tier and lower search servers. This method of targetinga video query to specific databases likely to have videos with similarclassification reduces the cost of searching by orders of magnitudewhile losing little or no accuracy.

Classification for search partitioning by database partitioning on eachserver can be used to partition the local databases at an index level oneach server. This method of partitioning database by index based onclassification also reduces the cost of searching by orders of magnitudewhile losing little or no accuracy. The encoded bit representation ofthe classification information is used to speed up search operations, byutilizing these encoded bit representations are used as index lookupaddresses for quick access to both query and reference information, forexample. These encoded bit representations are a compact form ofinformation useful for a targeted search in particular databases. Thismethod relies on further sub-dividing the database for first search stepby using the classification bits in the index.

System users can decide to use classification at any search level, fromclient search, to tier1 search, or tier2 search and so forth based onconsiderations of the cost and tolerable reduction in accuracy.

The method 700 of FIG. 7 begins at step 701, where the descriptors andsignatures available for a given video clip and within time frame foraudio signatures are collected. Other known or identified informationabout the content can be used to classify the video clip. Thisinformation can include an identified program logo, a user's history ofrecent viewing or statistics of user's queries, user profile, andcontent identified from an EPG. At step 702, distance measures are takenfor the given descriptors and signatures to previously identifiedclasses and their descriptors and signatures. The nearest classes thatmatch, by having the smallest distance metric, are identified in step703. The identified classes are used to limit search to certain parts ofthe database and speed up the search. At step 704, the query signaturesare submitted with the class id(s) which are represented as encoded bitsto a search operation. The additional class bits are used to narrow thesearch operation in the large databases.

FIG. 8A illustrates a method 800 of organizing a search database for ascalable content search system in accordance with the present invention.In FIG. 8A, content is organized into various search servers for a largescale content search system. Content is categorized into popular contentand important content that needs to be identified. Live or real timebroadcast content is received from live streams, fingerprinted, andadded to the content database. A subset of popular content, selectedcontent and selected live content are fingerprinted and added to adatabase of the first tier. As illustrated in FIGS. 2E and 2F, the firsttiers present significant performance gains by having a small databaseand fast updates. Hence it is useful to be able to update many of themost likely viewed channels in the first tier database, and often in oneor few servers. Hence, update rate is useful in determining the numberof servers required to update a specified and possibly required numberof broadcasts and live streaming channels.

The method 800 for organizing a search database utilizes databasemanagement techniques that include processing of popular content at step803, fingerprinting and adding associated metadata in step 804, andadding the fingerprints and the associated metadata to a search serverpopular content database in step 815. The popular content will thus beadded to a given search tier, often the second tier. In some cases, itis determined that the first tier will include some select popularcontent and the popular live broadcast and streaming channels since livebroadcast constitutes a majority which may be at times almost 80% of TVviewers. A good portion of the popular content will then be stored inthe second tier and the less popular content in lower tiers. Selectedlive broadcast content from radio or internet or cable and selectedpopular content is processed at step 801, fingerprinted and combinedwith metadata in step 805, and added to the first tiers search serversat step 814. All other relevant content is processed at step 802including gaming, movies, TV shows, internet programs, advertisement.Such content can be added to other tiers if statistically determined tobe relevant such as occurring more than a threshold of ten queries aday, for example. However, such content is generally added afterfingerprinting and metadata addition in step 806 to the lower searchtier servers at step 820, to support searching baseball content at step821, searching classical music content at step 822, talk show content atstep 823, and the like.

A not identified (NID) database of queries is collected during searchoperations and added typically to the first or second tiers, whereincontent identified to be more frequent or more likely is added to thefirst tier and content identified to be less likely is added to thesecond tier. The NID database is added to search servers in step 816.

FIG. 8B a method 830 of identifying the not identified queries (NID)queries and of creating the not identified database (NID DB) referencedatabase generated by these queries. The method 830 tracks queries thatare not identified, which are the queries for which no match is found inthe reference database, and creates a reference database for these NIDqueries to be searched for other queries. The motivation behind storingthe NID queries into the NID DB is that some content may become suddenlypopular due to a large interest triggered by some temporal cause.However, this content may have not been received and processed in thereference database. Such content could be captured by storing suchqueries to the reference database. Of course not all the NID queries canform a legitimate reference database hence a mechanism to remove suchless important NID queries from the reference database is utilized. Atstep 831, a query is received and searched in the search system. At step832, a determination is made whether a match is found. If a match isfound, then the process 830 proceeds to step 833. At step 833, theresults are returned to the remote client device. If the match is notfound, for example, the query is not identified, then the process 830proceeds to step 834. At step 834, the query is searched across an NIDdatabase. At step 835, a determination is made whether a match is foundin the NID database. If a match is found the process 830 proceeds tostep 836. At step 836, the results are returned to the remote clientdevice. However, if match is not found, the process 830 proceeds to step837. At step 837, the query is added to the NID database and then theprocess ends. When the query is added to the NID database at step 837,the search system can request the query client to send more information,such as the program name, channel or the uniform resource locator (url)of the content associated with the query to be stored as metadata in theNID database. The NID database may be maintained using cachingstrategies, such as Least Recently Used (LRU) or Least Used to keep theNID database size under a predetermined limit. The search operation atstep 831 may consist of online accesses by client queries and offlineaccesses for content management purposes in the search database and maybe carried out by a single tier or multiple tiers. The NID database mayitself be organized in various ways described in this patent or in theU.S. patent application Ser. No. 13/102,479 titled “Scalable, Adaptable,and Manageable System for Multimedia Identification” filed 6 May 2011.

FIG. 8C illustrates a method 850 of organizing a search database withclassification for a scalable content search system in accordance withthe present invention. In FIG. 8C, content is organized afterclassification of content for the various search servers for a largescale content search system. Each query content received is assigned alist of matching classifications so that the received queries can bedirected to specific databases associated with these classes with a highlikelihood of correct assignment. Classification of the content isperformed using signatures, metadata, and content metadata such asinformation concerning studio, actors, and the like. Classificationenables targeting of queries to specific search servers thus reducingsearch costs for very large databases. Alternately, classification isperformed not across servers but added to individual signature indexes,as described for the process 210 of FIG. 2B.

Live or real time broadcast content is received, fingerprinted, andadded to the appropriate database from live streams to the contentdatabase. A subset of popular content, selected content, and selectedlive content are fingerprinted and added to the appropriate database ofthe first tier. As presented in FIGS. 2E and 2F, the first tiers canpresent significant search performance gains by having a small databaseand fast updates so that as many of the most likely viewed channels arein the first tier database.

The method 850 for database management includes processing the popularcontent at step 853, fingerprinting and adding metadata at step 854, andadding signature and metadata database to search server at step 859. Thepopular content is added to a given search tier, often the second tier.Selected live broadcast content from radio or internet or cable andselected popular content is processed at step 851, fingerprinted andcombined with metadata at step 855, and added to the first tiers searchservers, at step 857. Other relevant content, including gaming, movies,TV shows, internet programs, advertisement is processed at step 852 andcan also be added to other tiers if statistically determined to berelevant such as occurring more than a threshold of ten queries a day,for example. Such content is generally added after fingerprinting andmetadata addition, at step 856. After classification at step 860, thequery is directed, at step 861 to the lower search tier servers tosupport searching baseball content at step 864, searching classicalmusic content at step 862, talk show content at step 863, and the like.

A not identified (NID) database of queries is collected during searchoperations and added typically to the first or second tiers, wherein themore likely content is added to the first tier and lesser likely contentadded to the second tier. The NID database is added to search servers instep 858.

FIG. 8D illustrates a method 880 of managing a not-identified databasefor a scalable content search system in accordance with the presentinvention. In FIG. 8D, the not identified (NID) queries are managed andutilized in a large scale content search system. FIG. 8D alsoillustrates how incoming queries that do not have a match (NID) areaccumulated and selected for storage and further use. The accumulatedNID queries are compared against a database of content to identify if itmatches any. Based on query frequency statistics any NID queries thathave matching content will be added to search system. If the NID queriesdo not match any content they will be added to the NID database in thesearch system. This mechanism of NID content management improves theaccuracy of the system and also reduces search cost and improves searchsystems performance.

The method 880 for NID database management includes at step 881,accumulating not id queries and maintaining the query statistics. Atstep 882, an aging process is used that eliminates less frequentqueries. One such method for elimination is removing the less frequentqueries and the queries that have not matched recently. At step 883,high volume queries are selected. At step 884, other relevant content,including gaming, movies, TV shows, internet programs, advertisements,youtube videos, user generated videos, viral ads, and the like arefingerprinted and added at step 890 to search servers used to matchagainst the NID content. If this NID content database is large relativeto the search database accessed by content queries, it is most likelyused offline and not on the search system accessed by remote clientqueries. In offline use, the NID content database is used morespecifically for the purpose of identifying NID queries. At step 892,the process 880 performs queries using the NID queries to the NIDcontent database search system. At step 894, a determination is madewhether there is a match. If there is no match, the process 880 proceedsto step 897. At step 897, the process 880 adds the NID query signaturesto the NID database in the main online search system. If there is amatch, then the process 880 proceeds to step 898. At step 898, thesignatures and metadata for the matching content, or sections thereof,are added to a search server in the main online search system. In thetiered search system, for example, the matching content of the NIDqueries could be added to the popular content search tier or to thelower search tiers. The query statistics for a given NID query can beused to decide the destination of the matching content.

FIG. 9A illustrates a method 900 of providing a customized overlay menuto a remote user for use with a remote search and tracking system inaccordance with the present invention. The method 900 provides a remoteuser with the benefit of a customized overlay menu while using remotesearch and tracking. A remote client device identifies content viewedremotely and may download additional reference signatures and relatedmetadata. The metadata describes, in addition to content information,and other links such as a uniform resource locator (url) for a scorecardfor the sports game that is being viewed, or url for a chat thread forthe program content or more specifically at the time location in theprogram, various menu options associated with the matching content sothat a user can view the content more meaningfully. For example, at anypoint in the media program the user can select a menu option to controlchannel selection and select more information. For example, menu optionsmay include a first selection to go to a next highlighted program, asecond selection to get more information about a show, actors, scene andthe like, a third selection to view user comments from social network onsecond screen, and the like. The choice of menu option can directlycontrol the user source device, such as a digital video recorder (DVR),a digital video disc (DVD) player, or the like. As the user continues toreceive media content, the remote search and tracking system follows thecontent and offers menu options associated with the identified content.

At step 903, a remote client performs fingerprinting on remote contentthat has been received, and at step 904 performs local search or searchon a central server. At step 901, download servers send signature andmetadata to the remote client to be stored on local signature databases.At step 906, the remote client tracks the incoming query content againstthe locally downloaded and stored signature databases at step 907, andmakes requests, on path 902 to the download servers if necessary.

At step 908, the remote client processes metadata links for identifiedtime location of scheduled media content, such as a scheduled TV show,and generates relevant menu options for the user. At step 909, thegenerated menu options are displayed. The menu options can includeoptions to skip to particular time locations in a program, or to skip toa set of program highlights or lists, or to provide other informationabout the program such as actors, scenes, directors, backstage shots, orto link to social network generated information. The user is able toselect these options and enabled to view only program highlights, or getsocial network comments on the show on a second screen application. Atstep 911, the user selects the option and is then able to directly andtransparently control the source device which may be a set top box,buddy box or DVR, or DVD player, via step 913. While the user is movingbetween different points in a given program, the remote search and tracksystem tracks the played content at steps 910 and 912, and is able toprovide the relevant menu options to the user.

In another example, a person watching a recorded program or DVD couldwatch a curated version such as highlights of program, or a PG-13version, or R rated version, or a G rated version of the same program byselecting the menu options on a second screen or remote or a secondscreen device doubled up as a remote device. Additionally, the usercould also select to choose menu options where they can get additionalinformation about the movie or scene or engage in a social network chator message thread related to the baseball game they are currentlywatching.

Overall the search and tracking function behaves as a silent observerand provides the user with choices for viewing on the TV screen or amain screen or a second screen. The choices for viewing can use apre-programmed set of edited content, and social channels forcommunication, and generic information sources.

FIG. 9B illustrates a signature data structure 920 for use in remotesynchronized applications. The remote data structure 920 includes thetime stamp 921, signature 922, content identification (ID) 923, featureinformation fields 924 and link to the menu options and metadata 925.The data structure 920 defines a content database element at a giventime in the reference. The content viewed by user aligns with a givencontent reference and a given time into that reference. The menuapplicable at that point is the nearest time stamp 921 in the array ofdata structures. Thus the time stamp 921 is used to identify therelevant menu option. The signature field 922, at a given time stamp921, and the signatures, typically more than one time instance and overa time range, are used to determine a match between query for remotelyviewed content and reference content databases. The content ID 923 maycontains a classification of the content based on class descriptors,suitable for directing a search to a classified search database. Thefeature information field 924 carries more information describing audioor video at the time index and time stamp 921. The additional fields,for example the feature information field 924, are useful in improvingthe accuracy of matching. The field 925 provides the link to menuoptions, at the given time index and time stamp 921.

FIG. 9C illustrates an exemplary menu data structure 930 suitable foruse in a metadata based data structure for remote synchronizedapplications in accordance with the present invention. The menu datastructure 930 is an array of menu options. In FIG. 9A at step 908, thedownloaded metadata information 907 is converted to the data structure930 so that it can used by a user device to control viewing ofinformation from multiple sources and also to be able to display andcontrol multiple viewing screens. The menu options are displayed on auser screen at step 909. At step 911, the selected menu option isexecuted. In each menu option, the field 931 provides the menudescription and display and the time parameter field 932 has the programtime link which can be used to directly skip to content or fast forwardor reverse to a particular point. An action parameter field 933 includesan action to be taken such as control the source device and the commandsto be sent to control other communication or content source devices.

The data structures and methods presented in FIGS. 9A, 9B, and 9C can beused to implement an embodiment of interactive selection, control anddisplay of content by the user, and presentation of menu options thatare context appropriate, after content has been identified as addressedherein, to the content viewed by user. The invention presents a methodfor publishing and interacting with video, audio, text content in anovel way by describing the control mechanism in a data structure asshown in FIG. 8B. The data structures 920 of FIGS. 9B and 930 of FIG.9C, enable publishing an existing DVD menu and program control,chapters, and enable user control on the user device, such as a tablet,laptop computer, smart phone, TV, or other such end user device. Aftercontent identification time points within the program can be identifiedand synchronized to by the user, the menu of control options for viewingthe content can be downloaded by the user to their smart device, ortracked on a server. Thus, a user can, for example, select differentoptions for viewing content on multiple screens. This invention bringstogether broadcast television or DVD, or cable and the internet, andinteractive control allowing users to use and interact with the contentin a novel and useful way. The user is thus able to watch and seamlesslycombine various sources, such as internet streaming, internet text,broadcast TV and the like, on a screen of choice. The curator ofcontent, which can be a producer of a program, or shared among expertusers or selected users, may generate multiple menu options that cancontrol the viewing with selected source control options, for example,and these multiple menu options form part of the metadata. Thegeneration of the original content and the metadata along with its menuoptions for control represents a unique and novel media publishingmethod. The metadata along with the content signatures can be downloadedby a user to enjoy this unique and flexible viewing experience.

It is understood that other embodiments of the present invention willbecome readily apparent to those skilled in the art from the followingdetailed description, wherein various embodiments of the invention areshown and described by way of the illustrations. As will be realized,the invention is capable of other and different embodiments and itsseveral details are capable of modification in various other respects,all without departing from the present invention. Accordingly, thedrawings and detailed description are to be regarded as illustrative innature and not as restrictive.

We claim:
 1. A method for content monitoring, the method comprising:organizing a reference database into a first tier database having searchcaches of popular and real time live content and long time storage ofthe popular and real time content and a second tier database havingcontent classified in separate databases; generating fingerprints in auser device for user watched content that includes popular and real timelive content; searching for the user watched content in a referencedatabase; generating a menu of option based on a reference databasecontent that matched with the user watched content; searching in thesearch caches for the user watched content and returning to the userdevice the menu of options if a match is found in the search caches;searching in the first tier long time storage if the user watchedcontent is not found in the search caches and returning to the userdevice the menu of options if a match is found in the long time storage;searching in the second tier database in a classified separate databaseaccording to a class of the user watched content if the user watchedcontent is not found in the long time storage and returning to the userdevice the menu of options if a match is found in the second tierdatabase; and displaying the menu of options on the user device tosolicit user selection of one displayed option.
 2. The method of claim1, wherein the search caches comprise: a found query cache storingcontent that matches frequent queries; and an unidentified query cachestoring queries that are not identified in the reference database. 3.The method of claim 1, wherein the user watched content is not found inthe reference database and the user watched content is stored in thereference database according to a class of the user watched content. 4.The method of claim 1 further comprising: responding to a user selectionof the one option according to an action parameter stored with theselected menu option in the user device; downloading informationconcerning the user watched content from a url specified with the actionparameter.
 5. The method of claim 1 further comprising: responding to auser selection of the one option according to an action parameter storedwith the selected menu option in the user device; controlling the userdevice based on the action parameter.
 6. The method of claim 1 furthercomprising: responding to a user selection of the one option accordingto an action parameter stored with the selected menu option in the userdevice; skipping to a particular time location in the user watchedcontent according to a time parameter and the action parameter.
 7. Amethod of performing search, the method comprising: storing selectedsubsets of reference databases containing classified reference contentsin a local cached database on a user device; classifying content of aquery generated on the user device to generate a classified query;counting signatures of the classified query that are separated fromother signatures in the classified query by a minimum distance togenerate a query unique signature count; performing a search with theclassified query upon determining the query unique signature countexceeded a predetermined threshold; comparing the classified query withthe classified reference contents stored in the local cached database onthe user device to perform the search with the classified query;generating a menu of option based on a reference database content thatmatched from the search with the classified query; and displaying themenu of options on the user device to solicit user selection of onedisplayed option.
 8. The method of claim 7 further comprising:determining to search a classified search database in a centralizedserver in response to determining the classified query did not find amatch in the local cached database.
 9. The method of claim 8, whereinthe classified search database is located on a search tier of acentralized server having a plurality of search tiers.
 10. A method forcontent monitoring, the method comprising: organizing a referencedatabase into a first tier database having search caches of popular andreal time live content; generating fingerprints in a user device foruser watched content that includes popular and real time live content;searching in the search caches for the fingerprints for the user watchedcontent and returning to the user device a menu of options if a match isfound in the search cache; and displaying the menu of options on theuser device to solicit user selection of one displayed option.
 11. Themethod of claim 10 further comprising: responding to a user selection ofthe one option according to an action parameter stored with the selectedmenu option in the user device; downloading information concerning theuser watched content from a url specified with the action parameter. 12.The method of claim 10 further comprising: responding to a userselection of the one option according to an action parameter stored withthe selected menu option in the user device; controlling the user devicebased on the action parameter.
 13. The method of claim 10 furthercomprising: responding to a user selection of the one option accordingto an action parameter stored with the selected menu option in the userdevice; skipping to a particular time location in the user watchedcontent according to a time parameter and the action parameter.
 14. Amethod of performing search, the method comprising: storing selectedsubsets of reference databases containing classified reference contentsin a local cached database on a user device; classifying content of aquery generated on the user device to generate a classified query;determining the information content of the classified query by using anentropy based information measurement on the classified query content;performing a search with the classified query upon determining theentropy based information measured exceeds a predetermined threshold;comparing the classified query with the classified reference contentsstored in the local cached database on the user device to perform thesearch with the classified query; generating a menu of option based on areference database content that matched from the search with theclassified query; and displaying the menu of options on the user deviceto solicit user selection of one displayed option.
 15. The method ofclaim 14 further comprising: determining to search a classified searchdatabase in a centralized server in response to determining theclassified query did not find a match in the local cached database. 16.The method of claim 15, wherein the classified search database islocated on a search tier of a centralized server having a plurality ofsearch tiers.